The invention relates to spectacles which, utilize a specular reflection on their interior side to record electronically a retinal reflex image formed in the eye. This image is modified by means of a computer and, by means of an illumination device and a back reflection, is physiologically without delay by way of the same spectacles superimposed on the original image such that an improved visual impression is created.
The use of opto-electronic spectacles for the reflection of computer-generated images into the eye, which has the name xe2x80x9ccyberspacexe2x80x9d or xe2x80x9cvirtual realityxe2x80x9d, is increasing very rapidly. This technique has a wide range of benefits for an application in the entertainment industry as well as in various fields of industry, traffic and medicine, and its spread and significance will constantly increase with the availability of increasingly faster image processing computers.
The application by means of closed non-transparent glasses, in the case of which images are provided to the eye by miniaturized cathode ray tubes or liquid-crystal matrices by way of mirror systems or glass fiber systems, is most common. It is a particular advantage of this technique to couple by means of a moving three-dimensional imaging, the image sequence or the action with different movements of the wearer of the spectacles. Thus, a changing of the viewing direction as the result of the movement of the head or the change of the perspective is simulated as the movement progresses. The movements of the arms and the fingers of the wearer of the spectacles can be entered into the image by means of sensors in order to permit him to directly intervene in the action.
In newer systems called xe2x80x9caugmented realityxe2x80x9d, the wearer of the spectacles, by means of partially transparent spectacles, can view the environment as well as an image of cameras, which is reflected in by way of the spectacles, of the same scene or of other image contents by means of a miniaturized monitor at the helmet. A well-known variant of this process which is called a xe2x80x9chelmet-mounted display (HMD)xe2x80x9d has already been introduced for the guidance of combat planes.
However, several problems have become known concerning this technique which are the result of the function of the sense of sight and await improved technical solutions. In the case of closed spectacles and a rigidly coupled monitor or monitor image, when the wearer of the spectacles moves his head, the scene moves along in the same direction, which unnaturally is in conflict with his visual habits. As the result of the imaging of the eye, he is accustomed to the fact that the scene extends precisely in the opposite direction. So far, this problem has been solved only incompletely by means of the cumbersome measuring of the head movement and of the eyeball by means of external angle-of-rotation sensors, by a corresponding image processing and by the tracking of the generated image.
As the result of the adaptation movements of the eyeball, which originate from so-called vestibular ocular reflexes (VOR) of the ear canal system and are used for holding the fixation point during movements of the head, the eye itself is capable of roughly stabilizing the retinal image. The fine adjustment takes place by means of the image as the reference. This image tracking is additionally used by the eye for adapting the VORs of a dynamic eye alignment.
This means that a superimposing of outside images can provide a realistic impression of the image only when they are coupled to the real retinal image.
In the case of closed spectacles, it is attempted to use the image of the blood vessels (ocular fundus) as the reference (retina tracking). However, this supplies an only insufficient resolution and is suitable only for monocular viewing (see, for example, E. Peli, xe2x80x9cVisual Issues in the Use of a Head-Mounted Monocular Displayxe2x80x9d, Optical Engineering, Vol. 29, No. 8, Page 883 (1990). A simultaneous stabilization in both eyes of images by means of these is virtually impossible because of the different alignment of the eyes. In addition to the deterioration of the image quality, the conflict between the vestibular and the visual information frequently leads to motor disturbances ranging to seasickness. These problems of the existing technology are described, for example, in the overview article by E. Peli, xe2x80x9cReal Vision and Virtual Realityxe2x80x9d in Optics and Photonics News, July 1995, Pages 28-34.
It is an object of the invention to solve the problems involving image stabilization in the case of the superimposing of outside images by means of the real image.
The invention is based on the older German Patent Application 19631414 with the title xe2x80x9cSystem for Detecting Retinal Reflexes and the Superimposing of Additional Images in the Eyexe2x80x9d. In this application, a system is described by means of which the retinal reflex image is detected by means of a confocally imagining, two-axis scanning system by way of the reflection of the interior side of partially transparent and correspondingly curved spectacles serially by means of a high-sensitivity photodetector.
It is suggested there to serially project the improved image on the retina by means of lasers and a beam splitter via the same light path in the reverse direction of the taken image.
Furthermore, it also becomes possible to additionally superimpose other images on the retina.
This technique basically solves the above-mentioned problems, but concrete implementations and applications are not indicated. It is the fundamental idea of the new invention to use this process for improving the perceptivity of the eye. The physical-technical problems which must be solved for this purpose are the result of the physiological characteristics of the eye and the constantly varying illumination conditions in the environment. Because of the variable light conditions and the different optical tasks, the eye is a very dynamic sense organ with respect to its basic functions. It adapts itself to the variation of the intensity of the background illumination for 12 decades. It changes from colored sight in daylight to purely black/white sight at night. Light in the wavelength range of from 400-1,500 nm is transmitted by the eye and imaged on the retina. In this case, only light in the range of from 400 nm to 750 nm is perceived; that is, the infrared light in the range of from 750 to 1,500 nm, which is very bright in the case of an exterior as well as an interior illumination, remains unutilized for the visual perception.
The eye horizontally and vertically covers an angular range of approximately 1000. However, the image resolution decreases very rapidly with the angular distance from the visual axis. The attentive momentary vision is limited to a central angular area of only +/xe2x88x925xc2x0, and the xe2x80x9csharpxe2x80x9d vision, for example, when reading or driving a car, is limited to the very small central angular range of +/xe2x88x920.5xc2x0. In addition, various movements of the eye also take place constantly. This results in consequences which, under certain circumstances, impair the perceptivity of the eye and are to be improved within the scope of the invention:
Adaptation
accommodation
sharpness capacity
defective vision
age-related reduced capability, and
movement dynamics.
It is an object of the present invention to suggest an arrangement which, similar to the eye, has very variably designed basic functions and is adapted to the requirements of the seeing process but simultaneously also takes into account and utilizes the special physiology and dynamics of the eye and the varying illumination conditions of the environment as well as the invisible IR-range. This can be achieved only insufficiently by means of the scanning variants (serial grid scan, serial spiral scan) indicated in the earlier application. This concerns the scanning pattern of the image-taking of the retinal reflex as well as the back-projection of the laser image into the eye.
A basic problem of the serial image scanning in contrast to the parallel image scanning is the short dwell time of the scanner in each image pixel. A uniform scanning of, for example, 0.5 million image points during a scanning time of 40 ms means an integration time of only 0,08 xcexcs, that is, 80 ns, in each image point. In comparison, the parallel time integration of all image points of the eye itself is 10-20 ms.
As known from the use of lasers for detecting the retinal structure of the eye in the so-called laser scanning ophthalmoscopes, a laser power of approximately 40 xcexcW is required in order to achieve during a laser scan a signal-to-noise ratio of 17 from one image pixel (see, for example, A. Plesch, U. Klingbeil and J. Bille, xe2x80x9cDigital Laser Scanning Fundus Cameraxe2x80x9d Applied Optics, Vol. 26, No. 8, Pages 1480-1486 (1987)). Converted to the larger surface, this would correspond to an intensity of irradiation in an image of an extensive source on the retina of 40 W/cm2, which corresponds to the intensity of irradiation of bright spotlights or the sun on the retina; that is, it is only by means of the grid scan, that relatively bright sources can be recorded on the retina with a good signal-to-noise ratio. In order to detect the imaging of weaker sources on the retina, the sensitivity must be increased significantly.
However, for detecting the retinal reflex, the serial image scanning has the decisive advantage of a better suppression of scattered light, of a simpler detecting lens system and of the possibility of an exact reversal of the beam path during the image back-projection by means of a laser and, for these reasons, should be retained also in this application. However, an extension of the dwell time can be achieved by changing the scanning pattern.
Because of the non-uniform distribution of the photo-receptors, with the highest density of the cones for the sharp vision in the center of the retina and the opposite course of the small rods for the out-of-focus but light-sensitive night vision, the grid scan is by no means the optimal scanning pattern. A scanning pattern which is adapted to the seeing process, for day vision, should become increasingly slower and denser in the direction toward the center; and for an adaptation to night vision, it should be precisely the reverse.
In addition to the dwell time, the received signal can be influenced by changing the spot size of the scan and thus also the image resolution.
The number of signal photons N, which are recorded by a scanning recording unit from the retina per image pixel can be calculated according to the following formula:
Ns=(BTxcex94xcexxcfx84)(AoR)(S/2xcfx80)(Ap/D2)(1/xcex5)
wherein
B=the spectral irradiation intensity on the retina
T=the optical transmission from the retina to the photo-detector
xcfx84=the integration time in an image pixel on the retina
Ao=the surface of the image pixel
R=reflectivity of the image pixel
xcex94xcex=spectral width of the receiving signal
Ap=pupillary surface
D=distance from pupilla to retina
S/2xcfx80=the angular distribution factor of the optical backscattering of the retina
xcex5=energy of a photon at the detecting wavelength
As illustrated by this formula, stronger signals, that is, a larger number of signal photons can be obtained by the following measures at the detecting instrument:
Extension of the dwell time xcfx84 of the scan in the individual image points,
enlargement of the scanning spot Ao on the retina,
enlargement of the spectral bandwidth xcex94xcex.
The invention suggests the scanning of the retina in a sequence of concentric circles (the center of the circle is equal to the fovea centralis) whose radius is enlarged and reduced successively. This type of scanning is called a circle scan. Because of the rotational symmetry of the eye lens and of the pupil about the visual axis and the rotationally symmetrical distribution of the photoreceptors in the retina, the circle scan is optimal.
The invention also suggests that an identical circle scan is used for the detection of the retina reflex from the environment and the image projection by means of the laser. Since, in the case of the circle scan from the outside to the center, after the reaching of the center, the scanning axis extends backward along the same path, the detection may optionally be used in the case of the scan to the center and the projection may be used from the center to the outside, or the detection may be used for the entire scanning process and the projection may only be used in a second one.
In the case of a constant deflection of scanning mirrors in two directions (Lissajou-Figur), during the circle scan, a slowing-down of the dwell duration necessarily takes place in the direction of the center. However, the invention provides that, for day vision, the scan duration of adjacent circles be additionally slowed down depending on the illumination conditions and can even be accelerated for night vision.
Because of the non-uniform distribution of the cones over the retina with a density in the center which is by more than two decades higher, the scanning rate (dwell duration per image point) in this area can be increased by this factor, 100.
For the night vision with a higher distribution of the small rods with an increasing radius, it is useful for the dwell time to decrease to a similar degree in the opposite direction toward the outside.
As known to the person skilled in the art, a circle scan can be implemented in an analog control by means of periodically swinging orthogonal scanning mirrors or in a digital control, by an approaching of the circular track with a large number of straight routes. The third alternative is the use of programmable algorithms of analog control signals which can be digitally called and are best suited for these variable conditions.
So that the receiving signal can be additionally increased also by the enlargement of the scanned image spot proportionally to its surface, the invention also provides that the momentary image pixel size on the retina can be variably adjusted in addition to the scanning rate.
With the change of the image spot size, the image resolution is also adapted corresponding to-the situation. In addition to changing the scanning surface, the resolution can also be adjusted by the variable radius jump of the scanning radii.
By means of an enlargement of the scanning image pixel of, for example, 10 xcexcm to 100 xcexcm, the image resolution, for example, of approximately 2 to 20 arc minutes (resolution range of reading and viewing) is reduced by a factor 10; the received signal is simultaneously increased by a factor 100.
As known to the person skilled in the art, during the confocal scanning, the image resolution is determined by the diaphragm diameter in the intermediate focus in front of the photodetector and can be adjusted by its change. The invention provides that liquid-crystal diaphragms or electro-optical diaphragms are used for this purpose, so that this adjustment can be carried out as rapidly as possible, that is, within one scanning cycle.
Since the time sequence of the scanning and the size of the image pixel during the detection and the projection should be as identical as possible, the invention suggests that the change of the scanning sequence and the diaphragm control in the projection channel be the same as in the detection channel. The variation of the optical integration time and the image pixel surface can then be compensated in the projection channel by the corresponding variation of the transmission power of the laser.
Furthermore, the level of the receiving signal depends on the spectral bandwidth of the receiver and can be increased by its widening. The invention provides that in the range of the bright daylight vision (photoptic vision), a splitting of the beam path into the color channels red-green-blue, each with a spectral width of approximately 100 nm, which corresponds to the color sensitivity of the eye, can be carried out. This permits a true-color image taking and, by means of corresponding three-color lasers, a colored back-projection into the eye.
In a low illumination of the environment, in which the colors are no longer perceived by the eye (scotopic vision), the invention provides the combining of all channels to one single (black/white) receiving channel without a color resolution. Furthermore, the invention provides that this receiving channel comprises not only the visible range of from 400-700 nm but, in addition, the close infrared range of from 700-1,000.
This has the following advantages for increasing the receiving signal in a low background illumination:
Between 400-1,000 nm, the eye has the full transparency and images a comparable image between 700-1,000 nm as between 400-700 nm;
the degree of reflection of the retina between 700-1,000 nm amounts to R=10-20% in comparison to R=3-5% between 400-700 nm;
photodetectors with a high quantum efficiency, such as photomultipliers and silicium avalanche diodes, are available over the whole spectral range of from 400 to 1,000 nm;
incandescent bulbs, which are used for the interior illumination of buildings, or outside, for the lighting of streets and in the case of vehicles, radiate 10 times more light between 700-1,000 nm than between 400-700 nm.
the reflectivity of the vegetation in nature is by a factor 5-10 higher between 700-1,000 nm than between 400-700 nm.
As illustrated by these examples, in low illumination (night vision), another increase of the receiving signal by a factor 100 can take by place by expanding the spectral range.
The expansion of the spectral range can either be fixedly installed in each instrument or can be designed to be variable by changing spectral filters. If a colored representation is not required, it is useful to use green laser light for the back-projection into the eye because of the extreme sensitivity and contrast perception of the eye with respect to this color.
Additional methods for the signal improvement, which can be used here, are the integration of several successive images and the image correlation, for example, images of both eyes.
On the whole, as the result of the variation of the two parametersxe2x80x94the dwell time of the scan in the image pixels and the size of the image spotxe2x80x94with the addition of the infrared range and the use of image correlation, the complete dynamics of the receiving signals can be detected over seven decades.
In the case of a complete optical transmission of the receiving channel of T=0.2 (see above formula), the receiving range of this dynamic detection system comprises irradiation intensities on the retina between 10xe2x88x925 W/cm2 and 100 W/cm2, which comprises the range of the typical inside and outside luminosity.
Because of the slow and fast eye movements, it is necessary to design the scanning system such that it constantly follows the change of the axis of vision through the spectacles; that is, that the axis of symmetry of the image scanning during the detection as well as during the projection, is identical with the axis of vision.
For achieving this object, the invention provides that a centering of the circle scan on the pupil of the eye is carried out before and after the scanning of the retinal reflex and the image projection into the eye. In this case, the largest scanning angle of the circle scan is selected such that, if the axis of symmetry of the scan deviates from the axis of vision, the exterior surface of the eyeball, the sclera with the iris and the opening of the pupil, is scanned by the circle scan. Since these parts of the eye, which are well illuminated by the exterior light, are not imaged-sharply but diffusely in the intermediate image plane of the photodetector, the receiving signal supplies no image information but an integral indication of the optical backscattering capacity of the original pattern.
If the receiving signals are compared with one another for equally long time sections, for example, quadrants, from each circle, they will only be of the same level if the axis of the circle scan is identical with the axis of the eye (vision axis). Because of the different backscattering from the sclera, the iris and the opening of the pupil, signal differences then represent a measurement concerning the amount of deviation of the axis and its direction. After a scaling of the entire signal by way of each circle, these deviation signals can be used for adjusting the zero position of a next circle scan (bias). Thus, an original deviation of the axes can be reduced with each circle scan, until it becomes imperceptibly small when the circle scan dips through the opening of the pupil (pupil tracking).
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of the invention when considered in conjunction with the accompanying drawings.